Skip to content

feat(conformance): add data-driven ACP core v1 suite with CI/nightly coverage#130

Merged
osolmaz merged 6 commits intoopenclaw:mainfrom
lynnzc:feat/acp-core-conformance-v1
Mar 13, 2026
Merged

feat(conformance): add data-driven ACP core v1 suite with CI/nightly coverage#130
osolmaz merged 6 commits intoopenclaw:mainfrom
lynnzc:feat/acp-core-conformance-v1

Conversation

@lynnzc
Copy link
Contributor

@lynnzc lynnzc commented Mar 13, 2026

Summary

Add a protocol-level ACP conformance suite for ACPX with a data-driven runner and CI/nightly coverage.

This establishes a stable, deterministic baseline for core ACP behavior and makes it straightforward to run the same profile against real adapters via --agent-command.

Why

ACPX currently relies on integration tests and manual checks to catch ACP regressions. What was missing is a protocol-focused contract layer that is deterministic, adapter-agnostic, and easy to extend.

This PR adds that missing layer.

Changes

  • Add conformance/spec/v1.md (normative v1 contract)
  • Add conformance/profiles/acp-core-v1.json (required-case profile)
  • Add 20 core conformance cases under conformance/cases/*.json
  • Add data-driven runner at conformance/runner/run.ts
    • JSON-defined steps execution (new_session, prompt, prompt_background, await_background, cancel, sleep)
    • JSON-defined checks evaluation
    • per-case permission_mode override support
    • JSON report output support
  • Add conformance/README.md usage and model docs
  • Add conformance:run script in package.json
  • Integrate CI smoke run in .github/workflows/ci.yml
  • Add nightly workflow .github/workflows/conformance-nightly.yml
    • full mock-profile run + artifacts
    • optional real-adapter smoke matrix (repo var / manual dispatch)

Validation

Local verification completed:

  • pnpm run conformance:run (20/20 passed)
  • pnpm run typecheck
  • pnpm run lint
  • pnpm run check:docs

Scope

Included:

  • ACP core lifecycle and baseline error semantics
  • deterministic protocol conformance baseline using test/mock-agent.ts

Not included in this PR:

  • provider-specific UX assertions
  • performance benchmarking
  • unstable ACP method coverage

@osolmaz
Copy link
Contributor

osolmaz commented Mar 13, 2026

great idea, will see if it needs rework and land if not

@osolmaz osolmaz self-assigned this Mar 13, 2026
@osolmaz osolmaz merged commit 01b6f1b into openclaw:main Mar 13, 2026
8 checks passed
@osolmaz
Copy link
Contributor

osolmaz commented Mar 13, 2026

Landed via temp rebase onto main.

  • Gate: GitHub CI green for this PR; local pnpm run check:docs passed; targeted conformance runner regressions passed; local pnpm run check missed only the repo-wide coverage threshold under Node v24.1.0 while the PR Test check passed on GitHub
  • Land commit: b36a3d8
  • Merge commit: 01b6f1b

Thanks @lynnzc!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants